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Executive  Summary 


As  steward  of  the  Capability  Maturity  Model®  (CMM®)  for  Software  (SW-CMM)  and  its 
related  products,  the  Software  Engineering  Institute  (SEI)  collects  information  related  to  the 
use  of  the  CMM  and  provides  feedback  to  the  user  community.  Information  collected  related 
to  assessment  findings  and  maturity  level  ratings  are  reported  to  the  community  in  the  Matur¬ 
ity  Profile  that  is  updated  twice  a  year.  The  Maturity  Profile  provides  a  current  status  of  the 
software  community  (reporting  organizations’  size  and  maturity  levels),  community  trends 
(growth  in  number  of  assessments  performed  and  shifts  in  the  maturity  profile  over  time), 
and  organizational  trends  (analysis  of  key  process  area  [KPA]  satisfaction  and  time  to  move 
up  in  maturity).  The  Maturity  Profile  summarizes  assessment  data  into  industry  aggregates 
and  can  be  viewed  on  the  SEI  Web  site:  http://www.sei.cmu.edu/sema/profile.html.  The  Au¬ 
gust  2001  Maturity  Profile  represents  data  from  1,483  CMM-Based  Appraisal  for  Internal 
Process  Improvement  (CBA IPI)  assessments. 


In  addition  to  information  that  contributes  to  the  Maturity  Profile,  data  discussed  in  this  re¬ 
port  are  collected  to  establish  the  usability  and  effectiveness  of  the  CBA  IPI  assessment 
method  in  order  to 

•  provide  feedback  to  the  community 

•  determine  the  user  satisfaction  (sponsors,  assessment  team  members,  and  Lead  Asses¬ 
sors)  in  the  assessment  method 

•  monitor  the  consistency  of  the  use  of  the  SETs  assessment  materials 

A  high-level  overview  of  the  CBA  IPI  method  is  published  in  a  technical  report  [Dunaway 
96b]  available  from  the  SEI  Web  site.  The  report  is  intended  for  executives  and  managers 
who  are  evaluating  or  planning  a  CBA  IPI  assessment  and  need  to  know  more  about  what  the 
assessment  entails.  The  report  is  also  intended  for  potential  assessment  team  members  and 
organizational  participants.  Detailed  guidance  for  conducting  a  CBA  IPI  is  contained  in  the 
Lead  Assessor’s  Guide  VI.  1  [Dunaway  96a]  and  is  available  to  those  persons  who  have  been 
accepted  into  CBA  Lead  Assessor  Training. 


Panelists  have  presented  their  experiences  using  the  CBA  IPI  assessment  method  at  domestic 
and  international  conferences  since  1996.  Five  of  these  panelists  contributed  to  a  technical 
report  published  in  1999  [Dunaway  99]  available  on  the  SEI  Web  site.  The  report  focuses  on 
the  organizational  perspective  before,  during,  and  after  the  assessment. 


Capability  Maturity  Model  and  CMM  are  registered  in  the  U.S.  Patent  and  Trademark  Office. 
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IX 


To  determine  user  satisfaction  in  the  assessment  method,  feedback  is  collected  at  the  conclu¬ 
sion  of  each  assessment  from  assessment  sponsors,  assessment  team  members,  and  the  as¬ 
sessment  team  leaders.  Results  from  feedback  were  reported  at  the  1996  Software  Engineer¬ 
ing  Process  Group  (SEPG)  Conference  [Dunaway  96c]  and  the  1998  SEPG  Conference 
[Dunaway  98].  This  current  report  contains  the  most  recent  information  that  has  been  col¬ 
lected  from  assessment  participants  and  is  a  follow-up  to  an  analysis  published  in  April  2000 
[Dunaway  00].  Details  of  questions  and  responses  are  in  Appendices  A,  B,  &  C. 


Assessment  sponsors  indicated  the  assessment  findings  were  “very  accurate”  in  reflecting  the 
state  of  the  software  process  in  their  organization.  More  than  85%  rated  the  assessment 
team’s  performance  to  be  “excellent.”  User  satisfaction  by  those  persons  who  requested  and 
funded  the  assessment  in  their  organizations  is  shown  to  be  very  good.  Eighty-one  percent  of 
team  leaders  agreed  with  the  sponsors  that  the  assessment  findings  were  “very  accurate.” 
Ninety-two  percent  of  team  leaders  rated  the  assessment  team’s  performance  either  “excel¬ 
lent”  or  “good.”  Team  leaders  and  team  members  reported  the  most  difficult  aspects  of  the 
assessment  were  “maintaining  a  realistic  schedule  for  the  on-site  period”  and  “consolidation 
of  data.” 


To  monitor  the  consistency  of  the  use  of  the  SEI’s  assessment  materials,  the  Lead  Assessor’s 
Requirements  Checklist  is  used.  The  form  is  intended  to  serve  as  a  quality  control  instrument 
as  well  as  to  assist  the  Lead  Assessor  in  keeping  track  of  the  implementation  of  each  of  the 
method’s  requirements.  Results  from  this  checklist  provide  metrics  to  the  community  for 
more  effective  planning  for  future  assessments.  Dunaway  first  reported  these  results 
[Dunaway  00].  This  report  contains  the  most  recent  information  that  has  been  collected  from 
the  Lead  Assessor  Requirements  Checklists. 


Sponsor  participation  in  the  Opening  Meeting  and  Final  Findings  Briefing  is  shown  to  be 
very  strong.  The  business  goals  of  the  assessment  are  being  identified  and  articulated.  Two- 
thirds  of  the  assessed  organizations  have  200  or  fewer  people  within  the  assessment  scope, 
that  is,  those  people  with  technical  and  managerial  responsibilities  for  software  development. 


The  Lead  Assessor’s  Requirements  Checklist  has  been  reported  to  be  useful  for  ensuring  that 
a  Lead  Assessor  follows  the  requirements  of  the  method  and  that  no  activity  is  omitted  or 
forgotten.  The  results  from  the  data  returned  indicate  a  reliable  adherence  to  the  method  re¬ 
quirements.  The  assessment  team  composition  and  training  are  being  accomplished  as  re¬ 
quired.  Tailorability  of  the  method  regarding  number  and  duration  of  interviews  appears  to  be 
adequate  while  satisfying  the  minimum  requirements. 
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Approximately  33  percent  of  the  assessment  on-site  activities  were  conducted  in  5  days. 
Forty-nine  percent  were  conducted  in  6  to  10  days.  The  median  numbers  of  hours  that  the 
assessment  teams  worked  together  (i.e.,  clock  hours)  for  the  assessment  activities  are 

•  Pre-On-Site  Activities:  37  hours 

•  On-Site  Activities:  62  hours,  which  includes  a  median  of  20  hours  for  consolidation  of 
data 

•  Total:  96  hours 

The  SEI  continues  to  collect  this  type  of  feedback  to  monitor  user  satisfaction  and  consis¬ 
tency  of  use  of  SEI  products  and  materials.  Additional  reports  will  follow  as  additional  data 
are  collected  and  analyzed. 
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Abstract 


As  steward  of  the  Capability  Maturity  Model®  (CMM®)  for  Software  (SW-CMM)  and  its 
related  products,  the  Software  Engineering  Institute  (SEI)  collects  information  related  to  the 
use  of  the  CMM  and  provides  feedback  to  the  user  community.  Assessment  data  are  reported 
in  industry  aggregates  by  the  SEI  in  the  Maturity  Profile,  which  provides  characteristics  of 
organizations  using  the  CMM  as  well  as  information  on  findings  and  maturity  levels.  The 
main  purpose  of  this  report  is  to  update  the  analysis  of  feedback  from  users  of  the  CMM- 
Based  Appraisal  for  Internal  Process  Improvement  (CBA IPI)  method  [Dunaway  00].  The 
assessments  for  which  data  are  contained  in  this  report  were  conducted  between  July  1998 
and  December  2000.  The  audience  for  this  document  is  the  community  of  managers,  execu¬ 
tives,  and  developers  who  are  planning  or  contemplating  having  a  CBA  IPI  assessment  in 
their  organizations,  assessment  team  members,  and  Lead  Assessors  who  are  interested  in 
learning  about  other  assessors’  experiences  in  order  to  improve  their  own  planning  and  use  of 
the  CBA  IPI  method. 


®  Capability  Maturity  Model  and  CMM  are  registered  in  the  U.S.  Patent  and  Trademark  Office. 


xiii 


CMU/SEI-2001-TR-021 


XIV 


CMU/SEI-2001  -TR-021 


1  Background 


Using  the  Capability  Maturity  Model®  for  Software  VI.  1  (CMM®)  as  a  reference  model 
[Paulk  93a,  Paulk  93b],  the  Software  Engineering  Institute  (SEI)  developed  the  CMM-Based 
Appraisal  for  Internal  Process  Improvement  (CBA IPI)  for  assessing  an  organization’s  soft¬ 
ware  process  capability.  CBA  IPI  V1.0  was  released  in  1995  and  updated  to  CBA  IPI  Vl.l  in 
1996  [Dunaway  96b]. 

The  SEI  has  published  reports  that  show  a  relationship  between  CMM-based  improvement 
and  organizational  performance.  Reports  indicate  improvements  in  cycle  time,  defect  density, 
and  productivity.  Benefit-to-cost  ratios  presented  range  from  4.0:1  to  8.8:1  [Herbsleb  94]. 
Results  show  that,  in  general,  increased  process  maturity  results  in  better  product  quality, 
ability  to  meet  schedule  commitments,  and  other  indicators  of  organizational  performance 
[Goldenson  95]. 

From  the  early  stages  of  assessment  usage,  the  SEI  has  collected  feedback  from  users  to  de¬ 
termine  how  well  assessments  are  working  and  how  to  improve  the  assessment  method.  The 
CBA  IPI  product  suite  includes  feedback  forms  to  collect  data  from  assessment  sponsors, 
team  members,  and  team  leaders.  Results  from  the  feedback  were  reported  at  the  1996  Soft¬ 
ware  Engineering  Process  Group  (SEPG)  Conference  [Dunaway  96c]  and  the  1998  SEPG 
Conference  [Dunaway  98], 


Questions  have  sometimes  been  asked  about  the  consistency  of  the  usage  of  the  CBA  IPI 
method.  Some  felt  that  Lead  Assessors  were  inteipreting  the  method  and  the  SW-CMM  vl.l 
model  in  different  ways;  for  example,  some  were  using  more  rigor  than  others  in  the  interpre¬ 
tation  of  the  model  as  well  as  execution  of  the  method.  In  an  effort  to  gather  data  on  the  con¬ 
sistency  of  assessment  usage  and  results,  an  additional  feedback  form  was  created:  the  Lead 
Assessor  Requirements  Checklist.  This  form  is  intended  to  provide  value  to  the  Lead  Asses¬ 
sor  in  planning  the  assessment  as  well  as  reporting  results  to  the  SEI.  The  form  focuses  a 
Lead  Assessor’s  attention  to  a  detailed  level  of  requirements  that  are  needed  for  a  CBA  IPI 
assessment.  Results  from  the  use  of  the  Lead  Assessor  Requirements  Checklist  were  first  re¬ 
ported  in  a  technical  report  [Dunaway  00]. 


Each  SEI-authorized  Lead  Assessor  who  conducts  a  CBA  IPI  is  required  to  return  certain  arti¬ 
facts  to  the  SEI  at  the  conclusion  of  each  assessment.  These  artifacts  reflect  assessment  data 


®  Capability  Maturity  Model  and  CMM  are  registered  in  the  U.S.  Patent  and  Trademark  Office. 
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that  will  be  incorporated  into  the  SEI  Maturity  Profile  to  provide  the  community  with  infor¬ 
mation  on  the  state  of  the  software  community’s  maturity,  and  data  that  will  assist  in  under¬ 
standing  how  well  the  assessment  method  is  working  and  provide  a  quality  control  mecha¬ 
nism  to  improve  consistency  of  use. 


Information  collected  related  to  assessment  findings  and  maturity  level  ratings  are  reported  to 
the  community  in  the  Maturity  Profile.  The  Maturity  Profile  provides  a  current  status  of  the 
software  community  (reporting  organizations’  size  and  maturity  levels),  community  trends 
(growth  in  number  of  assessments  performed  and  shifts  in  the  maturity  profile  over  time), 
and  organizational  trends  (analysis  of  key  process  area  [KPA]  satisfaction  and  time  to  move 
up  in  maturity).  The  Maturity  Profile,  which  is  updated  twice  a  year,  summarizes  assessment 
data  into  industry  aggregates  and  can  be  viewed  on  the  SEI  Web  site: 

http://www.sei.cmu.edu/sema/profile.html.  The  August  2001  Maturity  Profile  represents  data 
from  1,483  CBAIPI  assessments. 


In  addition  to  information  that  contributes  to  the  Maturity  Profile,  data  discussed  in  this  re¬ 
port  are  collected  to  establish  the  usability  and  effectiveness  of  the  CBA IPI  assessment 

method  to 

•  provide  feedback  to  the  community 

•  determine  the  user  satisfaction  (sponsors,  assessment  team  members,  and  Lead  Asses¬ 
sors)  in  the  assessment  method 

•  monitor  the  consistency  of  the  use  of  the  SETs  assessment  materials 

Data  to  help  understand  how  well  the  assessment  method  is  working  are  reflected  in  the  fol¬ 
lowing  feedback  forms: 

•  Sponsor  Feedback  Form;  requested  from  the  assessment  sponsor 

•  Assessment  Team  Leader  Feedback  Form;  required  from  the  team  leader 

•  Assessment  Team  Member  Feedback  Form;  required  from  each  team  member 

•  Lead  Assessor  Requirements  Checklist;  required  from  each  team  leader  to  provide  a  qual¬ 
ity  control  mechanism  to  ensure  that  each  of  the  requirements  of  the  method  has  been  ad¬ 
dressed 


Data  contained  in  this  report  are  obtained  from  the  following  sources: 


Type  of  Feedback  Form 

Number  of  Forms 

Dates 

Assessment  Sponsor 

72 

Sept.  1999  -  Oct  2000 

Assessment  Team  Member 

229 

Dec.  1999  -  Oct  2000 

Assessment  Team  Leader 

197 

Nov.  1998 -Oct  2000 

Requirements  Checklist 

260 

July  1998 -Dec  2000 
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There  were  780  CBA IPI  assessment  reports  received  by  the  SEI  and  incorporated  into  the 
Process  Assessment  Information  System  (PAIS)  between  July  1998  and  December  2000.  The 
Lead  Assessor  Requirements  Checklist  was  a  new  requirement  added  in  late  1998.  Although 
the  feedback  forms  are  stated  as  a  requirement  to  be  returned  within  30  days  of  the  conclu¬ 
sion  of  each  assessment,  the  return  of  these  forms  to  date  has  been  voluntary  and  not  enforced 
as  a  firm  requirement. 
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2  Assessment  Feedback  From  Participants 


Data  from  assessment  sponsors,  assessment  team  members,  and  assessment  team  leaders  are 
reported  in  this  document  from  CBA IPI  assessments  conducted  from  November  1998  to  Oc¬ 
tober  2000.  Analysis  of  prior  data  resulted  in  the  feedback  forms  to  be  updated  [Dunaway 
96c,  Dunaway  98].  More  questions  were  added  to  the  forms  for  team  members  and  team 
leaders  in  an  attempt  to  identify  the  most  troublesome  areas  of  the  assessment  method.  Data 
reported  here  are  from  the  updated  and  expanded  forms. 

2.1  Sponsor  Feedback  Summary 

The  responses  from  assessment  sponsors  are  detailed  in  Appendix  A.  Almost  78%  of  the  re¬ 
spondents  indicated  that  the  assessment  findings  were  “very  accurate”  in  reflecting  the  state 
of  the  software  process  in  their  organization.  More  than  85%  of  respondents  rated  the  as¬ 
sessment  team’s  performance  and  the  Lead  Assessor’s  performance  to  be  “excellent.” 


When  asked  how  well  the  assessment  findings  provided  guidance  for  planning  follow-on 
process  improvement  activities,  all  of  the  responses  were  either  “excellent”  or  “good.”  When 
asked  how  confident  they  are  in  being  able  to  actively  support  the  implementation  of  im¬ 
provements  based  on  the  assessment  findings,  82.6%  said  “very  confident”  and  17.4%  said 
“some  confidence.” 

2.2  Team  Leader  Feedback  Summary 

Responses  from  the  assessment  team  leaders  are  detailed  in  Appendix  B.  Eighty-one  percent 
of  respondents  indicated  that  the  assessment  findings  were  “very  accurate”  in  reflecting  the 
state  of  the  software  process  in  the  organization.  Nineteen  percent  indicated  the  findings  were 
“mostly  accurate.” 


Team  leaders  indicated  that  the  interviewees  were  “always”  forthcoming  and  honest  in  pro¬ 
viding  information  to  the  assessment  team  in  65%  of  the  responses.  Thirty-four  percent  indi¬ 
cated  the  interviewees  were  “frequently”  forthcoming  and  honest.  Seventy-one  percent  of  the 
respondents  indicated  the  assessment  team  members  were  “always”  objective  in  performing 
their  responsibilities;  29%  indicated  “frequently”  objective.  Ninety-two  percent  rated  the  as¬ 
sessment  team’s  performance  as  either  “excellent”  or  “good.” 
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Eighty-four  percent  indicated  they  were  “very  confident”  that  the  organization  would  actively 
support  the  implementation  of  improvements  based  on  the  assessment  findings. 


The  most  difficult  aspects  of  the  assessment  method  were  reported  to  be 

•  Maintaining  a  realistic  schedule  for  the  on-site  period;  9.8%  indicated  “difficult”  or  “ex¬ 
tremely  difficult.” 

•  Consolidation  of  data;  1 1.3%  indicated  “difficult”  or  “extremely  difficult.” 

•  Reviewing  documents  effectively;  7.9%  indicated  “difficult”  or  “extremely  difficult.” 

Otherwise,  the  activities  were  reported  to  be  “not  difficult”  or  “challenging,  but  manageable.” 

2.3  Team  Member  Feedback  Summary 

Responses  from  assessment  team  members  are  detailed  in  Appendix  C. 


In  prior  feedback,  team  members  expressed  a  need  for  greater  understanding  of  the  CMM. 
However,  in  Appendix  C,  the  respondents  report  their  CMM  knowledge  to  be  44%  “excel¬ 
lent,”  37%  “good.”  With  19%  of  respondents  reporting  their  CMM  knowledge  is  either  “fair” 
or  “poor,”  the  team  members  are  still  expressing  a  need  for  greater  understanding  of  the 
CMM.  Although  there  has  been  anecdotal  reporting  of  instances  of  inadequate  team  training 
provided  for  CBA IPI  team  members,  responses  indicate  this  is  not  a  pervasive  problem.  Four 
percent  of  the  assessments  had  no  team  training;  76%  indicate  the  entire  team  was  trained. 
Other  responses  show  that  abbreviated  training  was  given  in  approximately  11%  of  the  cases. 
When  asked  how  well  the  team  training  prepared  them  to  serve  as  an  effective  team  member, 
replies  were:  56%  “excellent,”  41%  “good,”  and  3%  “fair.”  For  the  most  part,  team  members 
believe  that  team  training  prepares  them  to  be  effective  team  members. 


Team  members  indicated  that  interviewees  were  “always”  (69.6%)  forthcoming  and  honest  in 
providing  information  to  the  assessment  team.  However,  29.5%  indicated  that  interviewees 
were  “frequently”  forthcoming  and  honest.  This  response  could  indicate  that  the  team  mem¬ 
bers  suspect  some  gaming  by  the  interviewees.  When  asked  if  the  assessment  team  members 
were  objective  in  performing  their  responsibilities,  replies  were:  80.9%  “always”  and  18.5% 
“frequently.” 


The  most  difficult  aspects  of  the  assessment  method  were  reported  to  be 

•  Managing  site  logistics;  10%  indicated  “difficult”  or  “extremely  difficult.” 

•  Maintaining  a  realistic  schedule  for  the  on-site  period;  12.9%  indicated  “difficult”  or  “ex¬ 
tremely  difficult.” 

•  Consolidation  of  data;  12.8%  indicated  “difficult”  or  “extremely  difficult.” 
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•  Preparing  draft  findings;  9.4%  indicated  “difficult”  or  “extremely  difficult.” 

•  Reviewing  documents  effectively;  7.9%  indicated  “difficult”  or  “extremely  difficult.” 

•  Abstracting  from  notes  (low  level  of  detail)  to  higher-level  observations  and  findings; 
8.7%  indicated  “difficult”  or  “extremely  difficult.” 

Otherwise,  the  activities  listed  in  the  table  were  reported  to  be  “not  difficult”  or  “challenging, 
but  manageable.” 

2.4  Prior  Issues 

In  analyzing  feedback  from  earlier  reports,  respondents  requested  further  information  about 
tailoring  the  assessment  method  for  low  maturity  organizations  and  small  organizations. 
Therefore,  these  issues  were  added  to  the  team  leader  and  team  member  feedback  forms. 

Responses  from  team  members  for  difficulty  in  tailoring  the  method  for  small  (fewer  than  50 
software  developers)  organizations  were 

•  22  responses:  “not  difficult” 

•  13  responses:  “challenging  but  manageable” 

•  2  responses:  “difficult” 

•  1  response:  “extremely  difficult” 

•  112  responses:  “not  applicable” 

Responses  from  team  leaders  for  difficulty  in  tailoring  the  method  for  small  (fewer  than  50 
software  developers)  organizations  were 

•  35  responses:  “not  difficult” 

•  28  responses:  “challenging  but  manageable” 

•  133  responses:  “not  applicable” 

Seventy  organizations  out  of  the  246,  or  28%,  that  responded  to  the  question  indicated  the 
assessed  organization  consisted  of  50  developers  or  fewer  (which  would  explain  the  high 
number  of  “not  applicable”  responses). 

Responses  from  team  members  for  difficulty  in  tailoring  the  method  for  low  maturity  organi¬ 
zations  (e.g.,  initial  assessments)  were 

•  28  responses:  “not  difficult” 

•  18  responses:  “challenging  but  manageable” 

•  3  responses:  “difficult” 

•  1  response:  “extremely  difficult” 
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100  responses:  “not  applicable” 


Responses  from  team  leaders  for  difficulty  in  tailoring  the  method  for  low  maturity  organiza¬ 
tions  (e.g.,  initial  assessments)  were 

•  41  responses:  “not  difficult” 

•  20  responses:  “challenging  but  manageable” 

•  1  response:  “difficult” 

•  134  responses:  “not  applicable” 

Since  a  very  small  number  of  responses  indicated  that  tailoring  was  “difficult”  or  “extremely 
difficult,”  it  appears  that  the  method  can  be  satisfactorily  tailored  for  small  organizations  or 
for  initial  assessments. 

2.5  Summary  from  Assessment  Participants 

When  asked  if  the  assessment  findings  accurately  reflect  the  state  of  the  software  process  in 
the  assessed  organization,  sponsors  indicated:  77.9%  “very  accurate,”  20.6%  “mostly  accu¬ 
rate;”  assessment  team  leaders  indicated:  81%  “very  accurate,”  19%  “mostly  accurate.” 

When  asked  if,  in  general,  the  CBA IPI  method  was  well  performed  for  the  organization,  as¬ 
sessment  team  members  indicated:  71.4%  “strongly  agree,”  27.6%  “agree,”  1%  “disagree”  or 
“strongly  disagree.” 


Over  the  past  few  years,  we  have  seen  improved  assessment  performance.  There  is  a  definite 
learning  curve  associated  with  being  an  assessment  team  leader  or  an  assessment  team  mem¬ 
ber.  Sponsors  and  their  organizations  are  more  familiar  with  the  S  W-CMM  V 1 . 1  and  assess¬ 
ments,  so  their  knowledge  is  greater  and  their  expectations  are  more  realistic. 


One  of  the  most  effective  improvements  that  Lead  Assessors  have  accomplished  is  to  refine 
the  automated  tools  that  they  use  during  data  collection  and  consolidation.  Many  have  re¬ 
ported  using  Excel  spreadsheets,  Microsoft  Word  tables,  and  some  commercially  available 
tools.  Automation  is  a  great  benefit  in  managing  data  collection.  LCD  projectors  are  an  effec¬ 
tive  tool  in  displaying  a  team’s  progress  in  building  observations  and  findings. 
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3  Lead  Assessor  Requirements  Checklist 
Planning  the  Assessment 


This  part  of  the  document  is  organized  based  on  the  format  of  the  Lead  Assessor  Require¬ 
ments  Checklist.  The  major  sections  in  the  checklist  correspond  to  the  phases  of  the  CBA IP1 
method: 


•  Planning  the  assessment 

•  Conducting  the  assessment 

•  Reporting  results 

•  Additional  questions 


The  CBA  IPI  requirement  indicated  in  the  checklist  is  shown,  along  with  a  question  as  to  how 
this  requirement  was  implemented.  Responses  for  each  requirement  are  summarized. 


The  first  requirement  for  a  CBA  IPI  is  that  an  authorized  SEI  Lead  Assessor  must  lead  the 
assessment  team.  The  Lead  Assessor  returns  a  Lead  Assessor  Requirements  Checklist  along 
with  the  other  feedback  forms  to  ensure  that  the  minimum  requirements  for  a  CBA  IPI  are 
met. 


3.1  Assessment  Materials 

Lead  Assessors  are  required  to  use  the  most  current  SEI  materials  when  conducting  the  CBA 
IPI  method.  The  material  may  be  obtained  by  the  Lead  Assessor  in  a  single  assessment  kit  to 
be  used  for  one  assessment,  or,  alternatively,  in  a  quantity  kit.  A  Lead  Assessor  in  an  organi¬ 
zation  that  has  multiple  Lead  Assessors  may  purchase  a  quantity  kit.  Each  Lead  Assessor 
within  the  organization  may  use  the  quantity  kit  as  many  times  as  needed  within  a  12-month 
period  from  the  time  of  purchase. 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Material  for  each  assessment  must  be  purchased  from  the 
SEI. 

Material  for  this  assessment  was  obtained  via: 

•  Single  kit 

•  Quantity  kit 

CMU/SEI-2001-TR-021 
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Figure  1:  Source  of  Assessment  Materials 

Out  of  the  256  responses  to  this  question,  159  indicated  that  a  quantity  kit  was  used,  while  95 
indicated  that  a  single  kit  was  purchased.  Two  entries  indicated  assessment  materials  were 
obtained  through  other  contract  mechanisms  with  the  SEI. 

3.2  Team  Composition 

The  CBA IPI  method  uses  a  team  of  qualified  persons  to  conduct  an  assessment.  A  team  ap¬ 
proach  is  used  rather  than  using  one  or  two  individuals.  It  has  been  observed  historically  that 
a  team  of  people  can  more  effectively  achieve  the  support,  confidence,  and  buy-in  within  the 
assessed  organization  for  the  assessment  results.  Team  members  must  sign  a  confidentiality 
agreement  whereby  each  member  guarantees  that  no  information  heard  or  seen  during  the 
assessment  will  be  attributed  to  an  individual  or  a  particular  project. 


3.2.1  Team  Size 

The  assessment  team  must  have  at  least  four  members  including  the  team  leader. 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

The  team  shall  have  four  to  ten  team  members.  At  least 
one  must  be  from  the  organization  being  assessed. 

•  Total  number  of  team  members 

•  Number  of  team  members  from  the  assessed  or¬ 
ganization 

10 
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Size  of  Assessment  Teams 
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more 


Number  of  Team  Members 


Figure  2:  Size  of  Assessment  Teams 

The  data  indicate  that  the  team  size  ranges  from  4  to  16  members  with  1  team  having  16  team 
members.  Out  of  258  responses,  13  (~5%)  did  not  correspond  to  the  required  team  size  of  4 
to  1 0  members. 


For  assessments  with  fewer  than  the  required  number  of  four  team  members,  the  assessment 
is  recognized  in  the  PAIS  database  as  an  “Other  Appraisal  Method.”  The  upper  limit  of  ten 
team  members  is  not  considered  to  be  as  critical  as  the  lower  limit,  although  teams  with  more 
than  ten  members  are  more  difficult  to  manage  and  bring  to  consensus  than  if  they  were  lim¬ 
ited  to  ten  people. 


At  least  one  of  the  team  members  must  be  from  the  assessed  organization;  however,  there  are 
usually  several  team  members  selected  from  the  organization.  The  organizational  team  mem¬ 
bers  usually  have  the  responsibility  to  effectively  utilize  the  assessment  results  in  the  organi¬ 
zation’s  follow-on  activities  in  the  process  improvement  program. 
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Number  of  Site  Team  Members 


c 

0) 

E 

in 

</) 

a> 

in 

in 

< 


a) 

n 

E 

3 


70 
60 
50 
40 
30 
20 
10 
0 

1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16 


58 

1 

o 

CJ 

CD 

34 

25 

20 

15 

9  r-i  4  6  5 

:  _  □  □  0  n  1  M  0 

Number  of  Team  Members  from  Assessed  Organization 


Figure  3:  Number  of  Assessment  Team  Members  from  Assessed  Organization 

The  teams  did  not  appear  to  have  difficulty  in  meeting  the  CBA IPI  requirement  to  have  at 
least  one  member  of  the  team  from  the  assessed  organization.  There  are  some  teams  com¬ 
prised  of  all  members  from  the  assessed  organization. 

3.2.2  Team  Member  Selection  Guidelines 

The  team  members  must  satisfy  criteria  prescribed  in  the  method  to  ensure  that  qualified,  ex¬ 
perienced  people  will  serve  on  the  team. 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Team  members  must  meet  the  selection  guidelines.  (Se¬ 
lection  guidelines  are  listed  in  the  Lead  Assessor’s  Guide 
vl.l .) 

Upon  checking  credentials  of  assessment  team  members, 
how  would  you  rate  the  team’s  experience  level  against 
the  recommended  guidelines? 

Rate  on  a  scale  from  1  to  5: 

(1-do  not  meet  guidelines;  5-exceed  the  guidelines) 
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Team  Members  Meet  Selection  Guidelines 


Figure  4: 


Rating  of  Team  Members  Based  on  Selection  Guidelines 


For  this  question,  respondents  are  asked  to  rate  on  a  scale  from  1  to  5:  (1 — do  not  meet 
guidelines;  5 — exceed  the  guidelines)  as  to  whether  the  team  members  meet  the  selection 
guidelines.  Most  teams  do  not  appear  to  have  difficulty  finding  team  members  who  meet  the 
selection  guidelines.  About  36%  of  the  responses  indicate  that  the  teams  are  comprised  of 
members  who  exceed  the  guidelines  (rating  5).  No  teams  report  that  they  have  team  members 
who  do  not  meet  the  selection  guidelines  (rating  1  or  2). 

3.3  Assessed  Organization 

3.3.1  Business  Goals 

Any  process  improvement  initiative  needs  to  be  focused  on  the  business  goals  of  the  organi¬ 
zation  in  order  to  justify  the  amount  of  effort  that  is  involved.  When  looking  at  the  entire 
CMM,  it  is  clear  that  business  goals  and  measures  established  in  the  early  maturity  levels 
become  even  more  important  when  quantitative  management  of  development  processes  at 
maturity  levels  4  and  5  is  based  on  meeting  the  business  goals. 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

The  assessment  is  discussed  with  the  sponsor  to  under¬ 
stand  the  business  goals. 

The  business  goals  of  the  sponsor  were  determined  to  be: 
(please  describe) 
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Business-Related  Goals 
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Business-Related  Goals:  #  Responses 

1 .  Improve  product  quality  and  reliability  46 

2.  More  efficient  development  processes; 

improve  productivity,  efficiency,  effectiveness  30 

3.  Deliver  on  time;  more  predictable  schedules  18 

4.  Improve  time-to-market  16 

5.  Satisfy/meet  customer  quality  needs  and  commitments  13 

6.  Reduce  costs  of  developing  software  13 

7.  Reduce  costs  of  maintaining/ support  costs  of  software  12 

8.  Maintain/improve  competitive  position  in  market  12 

9.  Reduce  defects  6 

10.  Improve  risk  management  4 

1 1 .  Deliver  on  budget  3 

12.  Reduce  cycle  time  3 

13.  Improve  quality  of  life  in  workplace  3 

14.  Improve  subcontractor  process  2 

15.  Reduce  re-work  2 


Total  Responses  183 

Figure  5:  Business-Related  Goals 

Fifty  percent  of  respondents  indicated  the  first  three  of  the  above  goals.  Seventy-five  percent 
of  respondents  indicated  the  first  six  goals. 

About  50%  of  the  responses  to  this  question  related  to  process  improvement  goals  rather  than 
the  organization’s  business  goals.  Obviously,  process  improvement  is  expected  to  have  a 
positive  impact  on  the  business,  but  it  is  encouraged  that  the  business  needs  be  explicitly 
stated  so  that  process  improvement  does  not  appear  to  be  an  end  unto  itself. 
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Process  Improvement-Related  Goals 


10  11  12 


Process  Improvement-Related  Goals: 


#  Responses 


1.  Validate/verify/achieve  maturity  level  2  40 

2.  Indicate  areas  of  improvement  to  guide  direction  for  future  improvement  3 1 

3.  Determine/monitor  progress/current  state  24 

4.  Validate/verify/achieve  maturity/capability  level  3  20 

5.  Establish  baseline  for  process  improvement  program  20 

6.  Acknowledge  improvements  achieved  10 

7.  Generate  management  and  staff  buy-in  and  support  for  process  improvement  10 

8.  Establish  best  practices  8 

9.  Validate/verify/achieve  maturity/capability  level  4  7 

10.  Validate/verify/achieve  the  organization's  maturity  level  7 

11.  Satisfy  corporate  goals  for  process  improvement  6 

12.  Validate/verify/achieve  maturity/capability  level  5  2 


Total  Responses  185 

Figure  6:  Process  Improvement-Related  Goals 

Many  organizations  establish  process  improvement  goals  across  the  organization  based  on 
business  needs  and  objectives.  The  business  goals  for  establishing  corporation-wide  process 
improvement  goals  are  communicated  throughout  the  organization. 

3.3.2  Organization  Size 

A  very  important  aspect  of  an  assessment  is  defining  the  scope  of  the  organization  to  be  as¬ 
sessed.  For  a  large  organization,  the  part  of  the  organization  that  is  undergoing  process  im¬ 
provement  under  the  auspices  of  a  single  senior  manager  is  a  likely  candidate  for  an  assess¬ 
ment  due  to  management  support  and  resources.  Since  an  assessment  is  only  one  part  of  a 
process  improvement  initiative,  the  assessment  results  will  impact  the  process  improvement 
program  and  its  progress.  The  sponsor  of  the  assessment  is  usually  the  senior  site  manager 
who  has  the  resources  and  authority  not  only  to  fund  the  assessment,  but  also  to  carry  the 
process  improvement  program  forward  utilizing  the  assessment  results. 
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CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

The  organization  scope  including  selected  projects  and 
participants  must  be  determined. 

There  are  persons  in  this  organization  with  technical 

and  managerial  responsibilities  for  software  development. 

The  organization  scope  is  detemiined  to  be: 

Organization  Size 


Fewer  20  to  50  50  to  100  to  200  to  350  to  500  to  1000  to 

than  20  100  200  350  500  1000  1500 

Number  of  Software  Developers 


Figure  7;  Organization  Size 


The  number  of  persons  in  the  assessed  organization  with  technical  and  managerial  responsi¬ 
bilities  for  software  development  range  from  a  minimum  of  5  people  to  a  maximum  of  1500 
people  in  the  246  organizations  who  responded  to  this  question.  The  above  figure  shows  the 
number  of  organizations  that  fall  into  each  range  of  organization  size.  One-third  of  the  or¬ 
ganizations  have  fewer  than  1 00  people.  Two-thirds  of  the  assessed  organizations  have  200  or 
fewer  people  with  technical  and  managerial  responsibilities  for  software  development.  This  is 
consistent  with  the  August  2001  Maturity  Profile  that  states:  “Nearly  half  of  the  organizations 
reporting  size  have  100  or  less  software  personnel.” 

3.3.3  CMM  Scope 

The  second  aspect  of  the  assessment  scope,  beyond  the  organizational  scope,  is  to  identify  the 
part  of  the  SW-CMM  V 1 . 1  that  is  most  relevant  to  the  organization  being  assessed.  If  the 
sponsor  is  aiming  for  a  maturity  level,  each  KPA  within  a  maturity  level  must  be  included  in 
the  scope,  along  with  each  maturity  level  below  that  level.  If  a  maturity  level  is  not  desired, 
individual  KPAs  may  be  chosen  for  the  assessment  scope.  The  minimum  CMM  scope  is  one 
KPA.  Although  it  is  probably  not  a  feasible  business  decision  to  have  an  organizational  inter¬ 
vention  as  extensive  as  a  CBA IPI  for  one  KPA,  it  meets  the  minimum  CBA IPI  requirements. 
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CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

The  CMM  scope  (KPAs  to  be  examined)  must  be  de¬ 
termined. 

The  CMM  scope  (KPAs  to  be  examined)  is  determined  to 
be: 

Level  2:  RM,  SPP,  SPTO,  SSM,  SQA,  SCM 

Level  3:  OPF,  OPD,  TP,  ISM,  SPE,  IC,  PR 

Level  4:  QPM,  SQM 

Level  5:  DP,  TCM,  PCM 

CMM  Scope:  KPAs  Included  in  Assessments 
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Figure  8:  CMM  Scope 

Figure  8  indicates  that  Software  Subcontract  Management  is  the  only  KPA  that  is  frequently 
designated  as  “not  applicable”  within  maturity  level  2.  All  other  KPAs  are  investigated  for 
maturity  level  2  and  3  assessments.  Approximately  19%  of  respondents  indicated  the  addition 
of  maturity  level  4  KPAs,  and  9%  indicated  the  addition  of  maturity  level  5  KPAs. 

3.4  Training 

The  CBA IPI  method  requires  that  assessment  team  members  receive  training  in  the  reference 
model,  SW-CMM  V 1 . 1 ,  prior  to  receiving  the  CBA  IPI  Team  Training. 

3.4.1  CMM  Training 

The  SETs  Introduction  to  the  CMM  course  is  the  preferred  training  method  for  assessment 
team  members,  although  a  Lead  Assessor  may  substitute  equivalent  CMM  training.  A  Lead 
Assessor  who  either  offers  his/her  own  CMM  training  course,  or  waives  a  team  member’s 
participation,  must  satisfy  the  Waiver  Guidelines.  It  is  at  the  Lead  Assessor’s  discretion 
whether  a  team  member  should  be  excused  from  CMM  training  due  to  previous  training. 
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Waivers  should  be  used  sparingly.  It  is  important  for  the  entire  team  to  have  consistent  under¬ 
standing  of  the  CMM  prior  to  an  assessment. 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

All  team  members  must  receive  the  SEI’s  Intro  to 

CMM  course  or  equivalent. 

Was  the  SEI’s  Intro  to  CMM  course  used  in  its  entirety? 

_ yes  _ no 

If  not,  what  CMM  course  was  used?  (e.g.,  source,  instructor, 
date) 

CMM  Training  Provided  to  Team  Members 


Figure  9:  CMM  Training  Provided  to  Team  Members 

The  chart  above  shows  the  distribution  of  teams  that  received  the  SETs  CMM  training  versus 
those  who  received  equivalent  training.  For  the  teams  with  SEI  training,  the  assessment  team 
members  attended  the  three-day  licensed  Introduction  to  the  CMM,  or  an  abbreviated  two- 
day  course  using  a  CD  that  is  provided  by  the  SEI.  For  the  teams  with  non-SEI  training,  the 
type  of  CMM  training  provided  is  typically  a  corporate  course  that  was  developed  in-house. 
Otherwise,  the  training  may  come  in  the  form  of  third-party  courses,  or  from  courses  devel¬ 
oped  and  taught  by  the  Lead  Assessor. 


It  is  acknowledged  that  the  quantity  of  training  hours  does  not  determine  the  quality  of  train¬ 
ing,  or  the  quality  of  students’  understanding.  If  a  reliable  examination  were  available,  a  stu¬ 
dent’s  knowledge  could  be  more  accurately  evaluated.  However,  no  such  accepted  examina¬ 
tion  exists  at  this  time.  As  the  chart  below  indicates,  there  are  wide  variations  of  durations  of 
CMM  training  courses  provided.  Where  CMM  training  is  indicated  to  be  so  brief  that 
“equivalence”  is  questionable,  the  Lead  Assessor  is  contacted.  The  assessment  is  subject  to  be 
classified  as  “Other  Appraisal  Method”  if  it  does  not  meet  the  requirement  of  providing 
CMM  training  to  the  assessment  team  members. 
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CMM  Training 
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Figure  10:  Duration  of  CMM  Training  for  Team  Members 


CMM  training  that  is  fewer  than  10  hours  in  duration  will  cause  the  Lead  Assessor  to  be  con¬ 
tacted  for  further  information  regarding  the  satisfaction  of  the  Waiver  Guidelines. 


3.4.2  CBA  IPI  Team  Training 

The  CBA  IPI  Team  Training  provided  by  the  Lead  Assessor  is  to  be  presented  in  a  minimum 
of  2.5  days,  or  approximately  20  classroom  hours.  Lead  Assessors  are  permitted  to  add  their 
own  material  to  enhance  the  course  or  expand  the  exercises  beyond  those  provided. 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

All  team  members  must  receive  the  SEI’s  CBA  IPI 

Team  Training  course. 

Was  the  SEI’s  CBA  IPI  Team  Training  used  in  its  entirety? 
yes  no.  If  no,  describe  modifications: 

Dates  and  time  (number  of  hours)  allocated  for  Team  Train¬ 
ing: 

Planned: 

Actual: 

Did  you  supplement  the  Team  Training  with  your  own  ma¬ 
terial?  _ yes _ no.  If  yes,  what  areas  did  you  supple¬ 

ment? 

How  many  team  members  participated  in  CBA  IPI  Team 
Training?  If  any  team  members  were  waived  from 

team  training,  please  indicate  the  Lead  Assessor  and  the 
dates  that  this  team  member  received  prior  CBA  IPI  Team 
Training: 

Team  members  waived  (names,  prior  training,  instructor  & 
dates): 

CMU/SEI-2001-TR-021 
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CBA  IPI  Team  Training 


Figure  11:  Instruction  Time  for  CBA  IPI  Team  Training 


Lead  Assessors  are  given  discretion  to  waive  a  team  member  from  all  or  part  of  Team  Train¬ 
ing  if  the  team  member  has  completed  this  training  within  the  past  few  months.  In  addition  to 
understanding  the  mechanics  of  the  assessment  method,  Team  Training  provides  an  opportu¬ 
nity  to  establish  rapport  among  the  team  members.  This  rapport  must  be  accomplished  some 
other  way  if  an  assessment  team  member  is  waived  from  Team  Training. 
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Figure  12:  Students  in  CBA  IPI  Team  Training  Class 

Some  organizations  train  multiple  assessment  teams  at  the  same  time  if  they  are  preparing  for 
multiple  assessments  in  the  same  time  frame.  A  pool  of  assessment  team  members  may  be 
prepared  together  if  they  will  participate  in  an  assessment  within  the  next  few  months  follow¬ 
ing  training. 
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4  Lead  Assessor  Requirements  Checklist: 
Conducting  The  Assessment 


There  are  four  data  collection  sources  used  in  the  CBA IPI  method: 

1.  Instruments;  maturity  questionnaires,  project  and  organization  questionnaires 

2.  Documents;  policies,  procedures  and  work  products 

3.  Interviews;  individuals  and  groups 

4.  Presentations;  draft  findings  presentations 

4.1  Maturity  Questionnaires 

Maturity  questionnaires  are  administered  prior  to  the  on-site  period  of  the  assessment.  Re¬ 
sponses  to  the  maturity  questionnaires  provide  guidance  to  the  assessment  team  for  the  focus 
of  interviews  and  document  review.  The  maturity  questionnaire  is  usually  tailored  to  include 
questions  related  to  the  KPAs  within  the  assessment’s  CMM  scope.  According  to  the  CBA  IPI 
requirement,  maturity  questionnaires  must  be  administered  to  at  least  the  project  leaders  from 
the  projects  being  investigated  in  depth.  A  sponsor  may  choose  to  have  the  maturity  ques¬ 
tionnaire  administered  to  additional  people  in  the  organization  to  enhance  organizational  par¬ 
ticipation  and  buy-in. 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Administer  maturity  questionnaires  for  at  least  the  pro¬ 
ject  leaders  from  the  selected  projects. 

How  many  questionnaires  were  administered? 

The  histogram  below  shows  the  range  in  the  number  of  maturity  questionnaires  that  were 
administered  in  the  260  organizations  that  responded  to  this  question.  For  most  assessments, 
5  to  1 0  questionnaires  are  administered. 
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Questionnaires  Administered 
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Figure  13:  Number  of  Questionnaires  Administered 


4.2  Interviews 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Conduct  interviews: 

-  project  leaders  (individual  interviews) 

-  middle  managers  (group  interviews) 

-  functional  area  representatives  (group  interviews) 

Indicate  number  and  duration  of  each  type  of  interview: 

-  project  leaders: 

-  middle  managers: 

-  functional  area  representatives: 

Total  number  of  interviewees: 

Total  Number  of  Interviewees 
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Figure  14:  Total  Number  of  Interviewees 

4.2.1  Project  Leader  Interviews 

The  assessment  team  interviews  project  leaders  individually  to  receive  independent  informa¬ 
tion  on  specific  representative  projects  being  investigated  in  depth.  The  recommended  num¬ 
ber  of  project  leader  interviews  is  four. 
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Project  Leader  Interviews 
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Figure  15:  Number  of  Project  Leader  Interviews 

It  is  recommended  that  a  typical  project  leader  interview  be  scheduled  for  1.0  to  1.5  hours.  In 
Figure  16  below,  data  indicating  that  project  leader  interviews  were  2.5  hours  or  greater  could 
represent  a  misunderstanding  of  the  question  and  may  represent  the  total  time  spent  in  project 
leader  interviews. 


Duration  of  Project  Leader  Interviews 
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Figure  16:  Duration  of  Project  Leader  Interviews 


4.2.2  Middle  Manager  Interviews 

Middle  managers  are  interviewed  as  a  group  as  long  as  there  are  no  people  in  the  interview 
with  reporting  authority  to  another  member  of  the  group. 
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Middle  Manager  Interviews 
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Figure  1 7;  Number  of  Middle  Manager  Interviews 

It  is  recommended  that  a  middle  manager  interview  be  1.0  to  2  hours  in  length.  As  shown  in 
Figure  18  below,  it  is  unlikely  that  a  middle  manager  interview  extends  past  two  hours.  For 
two  hours  or  greater,  the  respondent  may  have  given  cumulative  time  for  multiple  interviews. 


Duration  of  Middle  Manager  Interviews 
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Figure  18:  Duration  of  Middle  Manager  Interviews 


4.2.3  Functional  Area  Representative  Interviews 

Functional  area  representatives  (FARs),  or  software  practitioners,  are  interviewed  in  a  group 
of  typically  6-10  persons  whose  job  responsibilities  have  some  relationship  to  each  other.  No 
one  with  reporting  authority  to  any  other  member  of  the  group  is  permitted  in  the  same  inter¬ 
view. 


24 


CMU/SEI-2001-TR-021 


Number  of  FAR  Interviews 
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Figure  19:  Number  of  FAR  Interviews 

As  shown  in  the  figure  below,  FAR  group  interviews  are  typically  1.5  to  2  hours  in  duration. 


Duration  of  FAR  Interviews 


Figure  20:  Duration  of  FAR  Interviews 


4.3  Documents  Reviewed 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Conduct  document  review  at  a  minimum  for  each  KPA 
goal  within  the  assessment  scope. 

Approx,  how  many  documents  have  been  reviewed? 

Documentation  must  be  examined  at  least  for  each  goal 
for  each  KPA  within  the  assessment. 

Documentation  was  examined  for: 

•  each  key  practice 

•  each  goal 

The  first  question  is  intended  to  get  an  indication  of  the  amount  of  documentation  that  was 
reviewed  during  the  assessment.  This  is  a  difficult  measure  to  evaluate  since  there  are  many 
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ways  to  count  documents.  However,  in  spite  of  the  wide  variation,  it  is  clear  that  document 
review  is  a  major  source  of  data  for  the  CBA IPI  method. 
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Figure  21:  Number  of  Documents  Reviewed 


The  second  question  related  to  documentation  is  to  determine  if  sufficient  documentation  was 
examined  to  satisfy  the  minimum  requirement  of  having  at  least  one  document  for  each  goal. 
Ninety-one  percent  of  respondents  indicated  that  documents  were  reviewed  for  each  key 
practice.  Nine  percent  indicated  that  documents  were  reviewed  for  each  goal  and  “most  key 
practices.” 


4.4  Level  of  Data  Collection 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Collect  data  for  each  key  practice  for  each  KPA  within 
the  assessment  scope. 

Data  was  collected  only  at  the  goal  level. 

Data  was  collected  for  each  key  practice. 

Data  was  collected  for  each  subpractice. 

Level  of  Data  Collection 


Figure  22:  Level  of  Detail  of  Data  Collection 
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The  CBA IPI  method  requires  that  data  be  collected  for  each  key  practice  within  the  assess¬ 
ment  scope.  This  verifies  that  the  data  obtained  are  at  a  sufficient  level  of  detail.  Some  re¬ 
spondents  indicated  multiple  levels  of  data  collection.  Collection  of  data  at  the  subpractice 
level  is  permitted;  however,  goal  ratings  are  based  on  decisions  made  on  data  at  the  key  prac¬ 
tice  level.  Only  one  assessment  respondent  indicated  that  data  had  been  collected  only  at  the 
goal  level.  This  does  not  provide  adequate  detail  for  the  purposes  of  the  assessment  method 
and  is  a  violation  of  the  CBA  IPI  minimum  requirements. 


4.5  Data  Corroboration 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Data  was  corroborated  coming  from  at  least  two,  inde¬ 
pendent  sources  at  different  sessions. 

The  entire  assessment  team  determined  that  each  observa¬ 
tion  was  valid  (accurate,  corroborated,  consistent). 

The  method  requires  that  all  data  be  corroborated  by  at  least  two,  independent  sources  ob¬ 
tained  in  different  data  gathering  sessions.  For  this  question,  the  response  was  “yes”  for  all 
260  assessments. 


4.6  Observations 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Each  key  practice  for  each  KPA  within  the  assessment 
scope  must  be  detennined  to  be  sufficiently  covered 
with  observations  crafted  from  data  collected. 

number  of  observations  were  recorded  (total). 

The  assessment  team  detennined  sufficient  coverage  for 
each  key  practice  for  each  KPA  within  the  assessment 
scope.  yes  no.  If  not,  please  explain. 

Data  collected  from  each  data  gathering  session  is  consolidated  into  observations  related  to  a 
specific  key  practice. 
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Figure  23:  Number  of  Observations 


Each  key  practice  within  the  assessment  scope  must  be  sufficiently  covered  with  information 
related  to  the  key  practice’s  implementation,  the  organization,  and  the  development  life  cycle. 
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The  wide  variation  in  number  of  observations  could  represent  different  scopes  for  the  as¬ 
sessment  (number  of  KPAs  being  investigated)  as  well  as  different  styles  and  preferences  of 
Lead  Assessors  and  assessment  teams. 


Sufficiency  of  Data  Coverage 
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Figure  24:  Sufficiency  of  Coverage 


4.7  Draft  Findings 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Conduct  draft  finding  presentations. 

number  of  draft  findings  were  presented 

(how  many)  draft  finding  presentations 

Draft  findings  are  crafted  from  observations  and  are  based  on  a  KPA’s  strengths  and  weak¬ 
nesses. 
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Figure  25:  Number  of  Draft  Findings  Presented 

Draft  findings  are  presented  to  those  persons  who  have  been  interviewed  in  order  for  them  to 
validate  the  correctness  of  the  information  that  the  assessment  team  has  heard  and  seen.  Draft 
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findings  are  usually  presented  to  three  groups  of  persons  who  have  been  interviewed,  typi¬ 
cally:  project  leaders,  middle  managers,  and  FARs.  Flowever,  the  Lead  Assessor  has  latitude 
to  vary  the  number  of  draft  finding  presentations  according  to  the  organizational  needs.  Draft 
findings  presentations  are  additional  data  gathering  sessions,  and  attendees  are  encouraged  to 
comment  upon  the  accuracy  of  the  draft  findings  at  this  point  in  the  assessment. 


Draft  Findings  Presentations 
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Figure  26:  Number  of  Draft  Finding  Presentations 


4.8  Type  of  Ratings 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Ratings  must  be  made  based  on  sufficiently 
covered  key  practices  mapped  to  the  KPA 
goals,  (maturity  level  rating  is  optional) 

Ratings  were  done  by  the  assessment  team  for: 

maturity  level 

all  KPAs  within  the  scope 

except  (KPAs  not  rated): 

each  goal  for  each  of  the  above  KPAs 

each  key  practice  within  each  of  the  above  KPAs 

(tailoring  option) 

Ratings  are  made  for  each  goal  for  each  KPA  within  the  assessment  scope.  Goal  ratings  are 
based  on  weaknesses  that  have  been  found  relative  to  each  key  practice  related  to  the  goal 
being  rated.  Maturity  level  ratings  are  optional.  Rating  of  key  practices  is  also  optional.  Goal 
ratings  are  made  independent  of  key  practice  ratings. 
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Ratings  Made 
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Figure  27:  Level  of  Ratings  Performed 


4.9  Decision-Making  Strategy 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Consensus  is  the  decision-making  strategy  of  an 
assessment  team. 

Decisions  were  made  by  consensus  of  the  assessment  team. 

All  respondents  indicated  that  the  teams  used  consensus  as  their  decision-making  strategy. 
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5  Lead  Assessor  Requirements  Checklist: 
Reporting  Results 


5.1  Sponsor  Participation 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

A  final  findings  briefing  must  be  given  to  the 
sponsor. 

The  sponsor  attended  the: 

_ Opening  Meeting 

Final  Findings  Briefing 

_ Executive  Session 

The  assessment  sponsor  is  encouraged  to  attend  the  Opening  Meeting  and  express  his/her 
encouragement  for  interviewees  to  view  the  assessment  as  the  organization’s  own  and  to  be 
forthright  and  honest  in  providing  information  to  the  assessment  team.  The  assessment  spon¬ 
sor  owns  the  assessment  results,  so  it  is  mandatory  for  the  sponsor  to  attend  the  Final  Find¬ 
ings  Briefing  to  hear  the  assessment  findings.  During  the  Final  Findings  Briefing,  the  sponsor 
usually  thanks  the  assessment  team  for  its  participation,  time,  and  hard  work,  thanks  the  or¬ 
ganization  for  its  participation,  candor,  and  cooperation,  and  gives  an  indication  of  how  the 
assessment  findings  will  be  used  for  the  process  improvement  initiative.  The  Executive  Ses¬ 
sion  usually  includes  the  sponsor  and  his/her  direct  reports  and  is  used  as  a  session  to  answer 
any  questions  relative  to  the  assessment  findings  and  to  discuss  follow-on  activities.  Confi¬ 
dentiality  and  non-attribution  continue  to  be  in  effect. 
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Figure  28:  Sponsor  Participation 
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5.2  Reports  Submitted 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

The  final  findings  briefing  along  with  the  KPA 
profile  must  be  submitted  to  the  SEI  within  30 
days  of  the  conclusion  of  the  assessment. 

The  following  are  being  submitted  to  the  SEI: 

PAIS  report  with  Organization  and  Project  Questionnaires 

Final  findings  briefing  with  KPA  profile 

Required  feedback  forms  (including  this  checklist) 

Assessment  plan 

The  CBA IPI  requirement  states  that  the  Final  Findings  Briefing  along  with  the  KPA  profile 
must  be  submitted  to  the  SEI.  The  responses  from  the  260  assessments  indicated  that  the  in¬ 
formation  was  returned  to  the  SEI  for  all  of  the  260  assessments.  Fourteen  out  of  the  260  as¬ 
sessments  indicated  that  they  did  not  submit  the  assessment  plan. 
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6  Lead  Assessor  Requirements  Checklist: 
Additional  Questions 


6.1  Length  of  Assessments 

In  the  beginning  of  the  Lead  Assessor  Requirements  Checklist,  the  start  and  end  dates  of  the 
assessment  are  recorded.  Based  on  these  dates,  the  number  of  days  for  each  assessment  can 
be  calculated  (ignoring  any  weekends  or  holidays).  The  following  histogram  shows  the  dis¬ 
tribution  of  the  number  of  assessment  days  for  the  260  assessments. 
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Figure  29:  Number  of  Assessment  Days  On-Site 


Out  of  260  assessments,  85  or  32.7%,  conducted  the  on-site  activities  in  5  days.  Only  28,  or 
10.8%,  indicated  that  more  than  10  days  were  used  for  the  on-site  period. 


There  is  an  inclination  to  attempt  to  complete  the  on-site  activities  during  a  5-day  period  of 
time  due  to  resource  availability.  This  is  satisfactory  when  only  a  portion  of  the  reference 
model  is  being  investigated,  for  example,  KPAs  for  maturity  level  2.  However,  when  the 
model  scope  exceeds  the  maturity  level  2  KPAs,  it  is  usually  necessary  to  extend  the  on-site 
period.  The  Lead  Assessor  and  the  assessment  team  schedule  the  on-site  period  based  on  sev¬ 
eral  variables: 

•  Scope  of  the  organization  being  assessed,  for  example,  number  of  interviews  planned 
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•  Scope  of  the  CMM  included  in  the  assessment,  for  example,  number  of  KPAs  to  be 
investigated 

•  Assessment  experience  of  the  team  members 

•  Other  organizational  considerations  that  might  affect  the  schedule 

6.2  Team  Hours 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

How  many  team-hours  (total  number  of  hours  the  team 
worked  together)  were  spent  in  pre-on-site  activities 
(e.g.,  team  training,  document  review,  scripting  ques¬ 
tions)? 

How  many  team-hours  were  spent  in  on-site  activities 
(e.g.,  interviews,  data  consolidation,  findings  prepara¬ 
tion  and  presentation)? 

How  many  team-hours  were  spent  in  data  consolidation 
activities? 

In  an  effort  to  identify  how  time  was  used  during  the  assessment  activities,  the  above  ques¬ 
tions  were  asked.  The  responses  for  these  questions  had  a  very  wide  variance.  We  inquired 
about  the  variance  from  a  large  sample  of  Lead  Assessors  who  responded  with  on-site  activ¬ 
ity  hours  greater  than  180,  which  would  represent  more  than  20,  8-hour  days  for  on-site, 
since  this  seemed  unlikely.  One  reason  for  this  wide  variance  is  that  there  are  different  inter¬ 
pretations  of  the  term  “team-hours.”  Most  responded  that  they  had  submitted  total  time  spent 
rather  than  elapsed  time.  Some  Lead  Assessors  interpreted  “team-hours”  to  mean  total  per¬ 
son-hours,  and  multiplied  the  number  of  team  members  by  the  time  spent  by  each  team 
member.  The  intention  of  these  questions  was  that  team-hours  refer  to  the  total  amount  of 
time  that  the  entire  team  works  together  on  team  activities  in  the  particular  phase  (pre-on-site, 
on-site).  Due  to  this  possible  ambiguity  in  the  data,  the  charts  in  this  section  represent  data 
where  on-site  hours  greater  than  180  have  been  omitted  or  recalculated  at  the  Lead  Assessor’s 
direction. 
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Figure  30:  On-Site  Activity  Hours 
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Consolidation  time  was  tracked  and  reported.  Since  consolidation  is  an  on-site  activity,  we 
compared  on-site  hours  to  consolidation  hours.  For  each  of  the  ranges  of  on-site  periods 
above,  an  average  of  31%  of  time  was  spent  performing  consolidation.  The  median  value  was 
33%  for  consolidation  as  a  part  of  the  total  time  for  on-site  activities. 


The  figure  below  shows  the  time  periods  to  perform  the  pre-on-site  and  the  on-site  activities, 
as  well  as  time  spent  in  consolidation  that  is  a  subset  of  the  on-site  activities.  The  chart  shows 
the  largest  observed  value  that  is  not  an  outlier,  the  smallest  observed  value  that  is  not  an  out¬ 
lier,  the  median  value,  and  the  25th  and  75th  percentiles. 
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Figure  31:  Hours  to  Perform  CBA  I  PI  Activities 
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7  Conclusion 


7.1  Summary 

When  asked  if  the  assessment  findings  accurately  reflect  the  state  of  the  software  process  in 
the  assessed  organization,  sponsors  indicated:  77.9%  “very  accurate,”  20.6%  “mostly  accu¬ 
rate;”  assessment  team  leaders  indicated:  81%  “very  accurate,”  19%  “mostly  accurate.” 

When  asked  if,  in  general,  the  CBA IPI  method  was  well  performed  for  the  organization,  as¬ 
sessment  team  members  indicated:  71.4%  “strongly  agree,”  27.6%  “agree,”  1%  “disagree”  or 
“strongly  disagree.” 


To  monitor  the  consistency  of  the  use  of  the  SEl’s  assessment  materials,  the  Lead  Assessor’s 
Requirements  Checklist  data  is  reported.  Sponsor  participation  in  the  Opening  Meeting  and 
Final  Findings  Briefing  is  shown  to  be  very  strong.  The  business  goals  of  the  assessment  are 
being  identified  and  articulated.  The  assessment  team  composition  and  training  are  being  ac¬ 
complished  as  required.  The  results  from  the  data  indicate  a  reliable  adherence  to  the  method 
requirements. 

Approximately  33  percent  of  the  assessment  on-site  activities  were  conducted  in  5  days. 
Forty-nine  percent  were  conducted  in  6  to  10  days.  The  median  numbers  of  hours  that  the 
assessment  teams  worked  together  (i.e.,  clock  hours)  for  the  assessment  activities  are 

•  Pre-On-Site  Activities:  37  hours 

•  On-Site  Activities:  62  hours,  which  includes  a  median  of  20  hours  for  consolidation  of 
data 

•  Total:  96  hours 

The  SEI  continues  to  collect  this  type  of  feedback  to  monitor  user  satisfaction  and  consis¬ 
tency  of  use  of  SEI  products  and  materials.  Additional  reports  will  follow  as  additional  data 
are  collected  and  analyzed. 

7.2  Key  Findings 

The  following  table  summarizes  some  of  the  key  findings  in  this  document  that  may  be  use¬ 
ful  references  for  Lead  Assessors: 
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Table  1:  Key  Findings 


Item 

Findings 

Planning  the  Assessment 

Team  size 

60%  responses  range  from  5-8  team  members 

Business  goals 

Top  3: 

Improve  product  quality  and  reliability 

More  efficient  development  processes/improve  productivity,  efficiency,  effec¬ 
tiveness 

Deliver  on  time;  more  predictable  schedules 

Process  improvement  goals 

Top  3: 

Validate/verify/achieve  maturity/capability  level  2 

Indicate  areas  of  improvement  to  guide  direction  for  future  improvement 

Determine/monitor  progress/current  state 

Organization  size 

66%  with  fewer  than  200  software  developers 

Less  than  2%  over  1000  software  developers 

CMM  training 

29%  SEI’s  Intro  to  CMM;  59%  other  CMM  training 

CBA  IPI  training 

Range  of  most  frequent  delivery  times:  16-32  hours 

Conducting  the  Assessment 

Maturity  questionnaires 

Range  of  most  frequent  number  of  questionnaires  administered:  5-10 

Interviewees 

Range  of  most  frequent  number  of  interviewees:  10  -  40 

Documents  reviewed 

Range  of  most  frequent  number  of  documents  reviewed:  100  -  150 

Observations 

Range  of  most  frequent  number  of  observations  created:  150  -  1000 

Draft  findings 

Range  of  most  frequent  number  of  draft  findings  presented:  20  -  40 

Range  of  most  frequent  number  of  draft  finding  presentations:  2 

Length  of  assessments 

Most  frequent  on-site  length:  5  days 

Team-hours 

Pre-on-site:  37  median 

On-site:  62  median 

Consolidation:  20  median 

Total  team-hours:  96  median 
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Appendix  A  Assessment  Sponsor 

Feedback  Data 


Each  Lead  Assessor  is  required  to  provide  the  assessment  sponsor  with  a  feedback  form.  The 
sponsor  is  asked  to  complete  the  form  and  return  it  either  to  the  Lead  Assessor  or  directly  to 
the  SEI.  The  following  responses  represent  data  from  72  forms  that  were  returned  by  assess¬ 
ment  sponsors. 

•  How  important  are  the  following  to  process  improvement  in  your  organization? 


Importance 


Great  <- . ALittle 


To  reduce  the  number  of  software  defects  found 
after  software  is  released 

56 

13 

1 

To  schedule  and  plan  a  software  project  more 
effectively 

58 

11 

1 

To  estimate  needed  resources  more  accurately 

40 

26 

2 

2 

To  satisfy  corporate  requirements  for  process 
improvement 

19 

29 

15 

7 

(The  above  table  indicates  number  of  responses,  not  percentages.) 

•  How  important  were  the  following  objectives  to  your  assessment?  Were  they  achieved? 


Importance 

Achieved? 

Great  . 

. ALittle 

Yes 

No 

To  initiate  a  software  process  improvement 
program 

28 

15 

8 

15 

57 

2 

To  monitor  progress  in  the  software  process 
improvement  program 

51 

11 

4 

4 

63 

1 

To  validate  the  organization’s  capability  matur¬ 
ity  level 

49 

12 

4 

4 

65 

1 

To  prepare  for  an  upcoming  Software  Capabil¬ 
ity  Evaluation  (SCE) 

10 

16 

8 

27 

35 

12 

(The  above  table  indicates  the  number  of  responses,  not  percentages.) 

•  Do  you  think  the  assessment  findings  accurately  reflect  the  state  of  the  software  process 
in  your  organization? 
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77.9%  Very  accurate 

20.6%  Mostly  accurate 

1 .5%  Somewhat  accurate 

Poor  reflection 

•  How  well  did  the  Lead  Assessor  set  your  expectations  for  the  assessment? 


Excellent 

Good 

Fair 

Time  required  of  team  members 

81.4% 

17.1% 

1.4% 

Time  required  of  organization  par¬ 
ticipants,  (e.g.,  interviewees) 

81.4% 

18.6% 

- 

Organization  disruption 

70.0% 

25.7% 

4.3% 

Context  of  results,  (e.g.,  KPA 
strengths  and  weaknesses) 

84.3% 

15.7% 

- 

•  How  would  you  rate  the  assessment  team’s  performance  for  this  assessment? 

85.5%  Excellent 

14.5%  Good 

Adequate 

Fair 

Poor 

•  How  would  you  rate  the  Lead  Assessor’s  performance  for  this  assessment? 


85.7% 

Excellent 

12.9% 

Good 

1.4% 

Adequate 

- 

Fair 

_ 

Poor 

•  How  much  involvement  did  you  have  in  determining  the  assessment  scope,  (both  the  or¬ 
ganization  scope  and  the  CMM  scope)? 

12.9%  I  left  it  to  the  discretion  of  the  Lead  Assessor  and/or  assessment 

team  members. 

7.1%  I  delegated  the  activities  to  another  person. 

24.3%  I  participated  some. 

55.7%  I  was  very  involved. 

•  How  well  do  you  expect  the  assessment  findings  to  provide  guidance  for  planning  fol- 
low-on  process  improvement  activities? 
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68.1%  Excellent 

31.9%  Good 

Adequate 

Fair 

Poor 

•  How  confident  are  you  that  you  will  be  able  to  actively  support  the  implementation  of 
improvements  based  on  the  assessment  findings? 

82.6%  Very  confident 

17.4%  Some  confidence 

Little  confidence 
No  confidence 
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Appendix  B  Assessment  Team  Leader 
Feedback  Data 


The  following  set  of  data  contains  information  returned  for  229  assessments.  Assessment 
team  leaders  are  asked  to  complete  the  following  questions,  and  the  following  data  were  re¬ 
ceived  in  the  responses: 

•  Do  you  think  the  assessment  findings  accurately  reflect  the  state  of  the  software  process 
in  the  organization? 

81%  Very  accurate 

19%  Mostly  accurate 

Somewhat  accurate 
Poor  reflection 

•  Were  the  interviewees  forthcoming  and  honest  in  providing  information  to  the  assess¬ 
ment  team? 


65% 

Always 

34% 

Frequently 

1% 

Sometimes 

- 

Rarely 

•  Were  the  assessment  team  members  unbiased  in  performing  their  responsibilities? 


71% 

Always 

29% 

Frequently 

1% 

Sometimes 

- 

Rarely 

•  How  would  you  rate  the  assessment  team’s  performance  for  this  assessment? 


55% 

Excellent 

37% 

Good 

7% 

Adequate 

1% 

Fair 

_ 

Poor 

•  How  confident  are  you  that  the  organization  will  actively  support  the  implementation  of 
improvements  based  on  the  assessment  findings? 
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84%  Very  confident 

15.5%  Some  confidence 

0.5%  Little  confidence 

No  confidence 

•  In  general,  the  CBA IPI  method  was  well  performed  for  this  organization. 


82% 

Strongly  agree 

17.6% 

Agree 

- 

No  opinion 

0.4% 

Disagree 

- 

Strongly  Disagree 

•  Some  areas  of  difficulty  had  been  reported  [Dunaway  98].  The  feedback  form  was  ex¬ 
panded  to  seek  more  specifics  on  where  the  difficulties  occurred.  The  following  table  re¬ 
ports  the  responses. 
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Table  2:  Team  Leader  Feedback 


Activity 

Not  difficult 

Challenging, 

but  manageable 

Difficult 

Extremely  difficult 

Not  applicable 

Team  Member  Training/Knowledge: 

Understanding  the  CMM  for  maturity  levels  2  and  3 

76.4% 

19.0% 

3.2% 

1.4% 

Understanding  the  CMM  for  maturity  levels  4  and  5 

23.3% 

10.5% 

1.4% 

0.5% 

64.3% 

Understanding  the  CBA  IPI  method 

80.2% 

16.1% 

0.9% 

- 

2.8% 

Planning: 

Managing  site  logistics  (e.g.,  facilities,  schedule) 

61.6% 

32.4% 

4.6% 

0.9% 

0.5% 

Maintaining  a  realistic  schedule  for  the  on-site  period 

46.5% 

42.8% 

8.4% 

1.4% 

0.9% 

Obtaining  team  members  who  meet  selection  criteria 

60.5% 

28.8% 

8.4% 

0.5% 

1.9% 

Time  Management: 

Conducting  interviews 

76.0% 

23.1% 

0.9% 

Consolidation  of  data 

47.9% 

40.6% 

8.8% 

2.3% 

0.5% 

Preparing  draft  findings 

59.4% 

36.4% 

3.2% 

0.5% 

0.5% 

Preparing  final  findings 

78.3% 

18.0% 

2.8% 

0.5% 

0.5% 

On-Site  Activities: 

Reviewing  documents  effectively 

50.5% 

41.2% 

7.4% 

0.5% 

0.5% 

Achieving  consensus  with  team  members 

59.4% 

33.6% 

5.5% 

1.4% 

- 

Identifying  the  appropriate  persons  in  the  assessed 
organization  for  interviews 

75.9% 

20.8% 

3.2% 

_ 

_ 

Developing  interview  scripts 

71.8% 

26.3% 

1.4% 

- 

0.5% 

Abstracting  from  notes  (low-level  of  detail)  to  higher-level 

57.1% 

40.1% 

2.3% 

0.5% 

- 

observations  and  findings 

Mapping  the  organization’s  practices  to  the  CMM 

75.0% 

22.7% 

1.9% 

0.5% 

_ 

Tailoring  the  Method: 

For  small  (<50  software  developers)  organization 

17.9% 

14.3% 

67.9% 

For  low  maturity  organizations  (e.g.,  initial  assessment) 

20.9% 

10.2% 

0.5% 

- 

68.4% 
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Appendix  C  Assessment  Team  Member 

Feedback  Data 


Assessment  team  members  are  asked  by  their  team  leader  to  complete  the  team  member 
feedback  form  supplied  in  assessment  kits.  One  of  the  team  members  is  asked  to  collect  the 
forms  and  send  them  to  the  SEI  instead  of  giving  them  to  the  Lead  Assessor.  This  is  to  pre¬ 
serve  confidentiality  of  the  responses  and  to  encourage  team  members  to  be  objective  about 
their  assessment  experience.  The  following  set  of  data  reflects  197  team  member  feedback 
responses: 

•  What  CMM  training  did  the  Lead  Assessor  provide  for  you  prior  to  the  assessment? 


•  How  would  you  rate  your  CMM  knowledge? 


How  Would  You  Rate  Your  CMM 
Knowledge? 


Fair  Poor 

16%  3% 


Excellent 

44% 


Good 

37% 


•  Did  your  software  experience  meet  the  team  member  selection  criteria?  A  response  of 
“yes”  was  provided  by  169  respondents,  and  “no”  was  given  by  2  respondents. 
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Did  the  Lead  Assessor  provide  CBA IPI  Team  Training  for  you? 


•  How  well  did  team  training  prepare  you  to  serve  as  an  effective  assessment  team  mem¬ 
ber? 


•  Were  the  interviewees  forthcoming  and  honest  in  providing  information  to  the  assess¬ 
ment  team? 

69.9%  Always 

29.5%  Frequently 

0.6%  Sometimes 

Rarely 

•  Were  the  assessment  team  members  objective  in  performing  their  responsibilities? 


80.9% 

Always 

18.5% 

Frequently 

0.6% 

Sometimes 

- 

Rarely 
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•  How  would  you  rate  the  assessment  team’s  performance  for  this  assessment? 


74.1% 

Excellent 

24.7% 

Good 

1.1% 

Adequate 

- 

Fair 

_ 

Poor 

•  How  would  you  rate  the  Lead  Assessor’s  performance  for  this  assessment? 


81.6% 

Excellent 

16.7% 

Good 

0.6% 

Adequate 

0.6% 

Fair 

_ 

Poor 

•  Some  areas  of  difficulty  had  been  reported  [Dunaway  98].  The  feedback  form  was  ex¬ 
panded  to  seek  more  specifics  on  where  the  difficulties  occurred.  The  following  table  re¬ 
ports  the  responses. 
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Table  3:  Team  Member  Feedback 


Activity 

Not  difficult 

Challenging, 

but  manageable 

Difficult 

Extremely  difficult 

Not  applicable 

Team  Member  Training/Knowledge: 

Understanding  the  CMM  for  maturity  levels  2  and  3 

60.7% 

35.8% 

2.3% 

0.6% 

0.6% 

Understanding  the  CMM  for  maturity  levels  4  and  5 

26.9% 

19.9% 

2.3% 

1.2% 

49.7% 

Understanding  the  CBA  IPI  method 

66.3% 

33.1% 

0.6% 

- 

- 

Planning: 

Managing  site  logistics  (e.g.,  facilities,  schedule) 

45.0% 

25.7% 

8.2% 

1.8% 

19.3% 

Maintaining  a  realistic  schedule  for  the  on-site  period 

40.9% 

35.1% 

11.1$ 

1.8% 

11.1% 

Obtaining  team  members  who  meet  selection  criteria 

58.0% 

15.4% 

3.0% 

0.6% 

23.1% 

Time  Management: 

Conducting  interviews 

58.7% 

34.3% 

7.0% 

- 

- 

Consolidation  of  data 

34.9% 

52.3% 

12.2% 

0.6% 

- 

Preparing  draft  findings 

49.1% 

41.5$ 

7.6% 

1.8% 

- 

Preparing  final  findings 

62.7% 

33.1% 

3.6% 

0.6% 

- 

On-Site  Activities: 

Reviewing  documents  effectively 

47.1% 

42.4% 

9.3% 

1.2% 

- 

Achieving  consensus  with  team  members 

56.4% 

37.8% 

5.8% 

Identifying  the  appropriate  persons  in  the  assessed 

organization  for  interviews 

54.5% 

24.0% 

5.3% 

1.2% 

15.2% 

Developing  interview  scripts 

46.5% 

42.4% 

5.8% 

1.2% 

4.1% 

Abstracting  from  notes  (low-level  of  detail)  to  higher-level 

46.8% 

43.9% 

8.1% 

0.6% 

- 

observations  and  findings 

Mapping  the  organization’s  practices  to  the  CMM 

54.7% 

40.7% 

3.5% 

1.2% 

- 

Tailoring  the  Method: 

For  small  (<50  software  developers)  organization 

14.7% 

8.7% 

1.3% 

0.7% 

74.7% 

For  low  maturity  organizations  (e.g.,  initial  assessment) 

18.7% 

12.0% 

2.0% 

0.7% 

66.7% 

•  In  general,  the  CBA IPI  method  was  well  performed  for  this  organization. 


71.4% 

Strongly  agree 

27.6% 

Agree 

- 

No  opinion 

0.5% 

Disagree 

0.5% 

Strongly  Disagree 
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